\documentclass[letterpaper]{article}
\usepackage{../../common/fspdoc}
\setcounter{secnumdepth}{5}
\setcounter{tocdepth}{5}
\lhead{{2015 \FSP{} Install Guide - {\small \em SLES12 Version}}}

% Include git variables
\input{vc.tex}

% Define Base OS and other local macros
\newcommand{\baseOS}{SLES12}
\newcommand{\LosF}{\emph{LosF*}}
\newcommand{\FSPVersion}{15.31}
\newcommand{\IMAGE}{sles12}

\newcommand{\install}{zypper -n install}
\newcommand{\chrootinstall}{zypper -n --root \$CHROOT install}
\newcommand{\groupinstall}{zypper -n install -t pattern}
\newcommand{\groupchrootinstall}{zypper -n --root \$CHROOT install -t pattern}

\begin{document}
\graphicspath{{../../common/figures/}}
\thispagestyle{empty}

% Title Page

{\hspace*{4in} \includegraphics[width=1.8in]{intel_1spot_100.pdf}}

\vspace*{2cm}
\noindent {\LARGE \color{RoyalBlue} \fontfamily{phv}\selectfont 2015 Forest Peak (v\FSPVersion{})} \vspace*{0.1cm} \\
\noindent {\LARGE \color{RoyalBlue} \fontfamily{phv}\selectfont Cluster Building Recipes} \\ 
\noindent\rule{6in}{0.4pt} \\

\noindent {\Large \baseOS{}* Base OS} \\ \vspace{0.2cm}

\noindent {\large {\em Base Linux* Edition }}

\vspace*{3in}

\noindent{\normalsize Intel Cluster Makers} \vspace*{0.1cm} \\
{\normalsize Copyright~{\small\copyright}~2014-2015 Intel Corporation} \vspace*{0.1cm} \\ 
{\normalsize Document Last Update: \VCDateISO} \vspace*{0.1cm} \\ 
{\normalsize Document Revision: \VCRevision} \\ \vspace*{0.1cm}

% Disclaimer  ----------------------------------------------------
\input{../../common/legal} 

\newpage
\tableofcontents
\newpage

% Introduction  --------------------------------------------------

\section{Introduction} \label{sec:introduction}
\input{../../common/install_header}
\input{../../common/intro} \\

\input{../../common/base_edition/edition}
\input{../../common/audience}
\input{../../common/requirements}
\input{../../common/inputs}

% Base Operating System --------------------------------------------

\section{Install Base Operating System (BOS)}
\input{../../common/bos}

% ------------------------------------------------------------------

\section{Install \FSP{} Components} \label{sec:basic_install}
\input{../../common/install_fsp_component_intro.tex}

\subsection{Enable \FSP{} repository for local use} \label{sec:enable_repo}
\input{../../common/enable_fsp_repo}

\begin{lstlisting}[language=bash,keywords={},literate={VER}{\FSPVersion{}}1]
[master]$ fsp_repo=http://tcgsw-obs.pdx.intel.com:82/ForestPeak:/VER/SLE_12_Intel/ForestPeak:VER.repo
\end{lstlisting}


If outside of Intel, the repository location should be set to match the path to
a locally accessibly copy of the \FSP{} repository. Once set, the repository can be
enabled via the following:

% begin_fsp_run
% fsp_comment_header Enable FSP repository for local use \ref{sec:enable_repo}
\begin{lstlisting}[language=bash,keywords={}]
[master]$ zypper addrepo ${fsp_repo}
[master]$ zypper -n --no-gpg-checks --gpg-auto-import-keys refresh
\end{lstlisting}
% end_fsp_run

% Dev testing for Factory
%[master]$ zypper addrepo $FSP_mirror/ForestPeak:/15.16/SLE_12_Intel/ForestPeak:15.16.repo


In addition to the \FSP{} package repository, the {\em master} host also
requires access to the standard distro repositories in order to resolve
necessary dependencies. For \baseOS{}, the requirements are to have access to
both the base and SDK repositories:

\begin{itemize*}
\item SLES12-12-0
\item SLES12-12-sdk
\end{itemize*}

\input{../../common/automation}

\subsection{Add provisioning services on {\em master} node} \label{sec:add_provisioning}
\input{../../common/install_provisioning_intro}
\input{../../common/firewall}

% begin_fsp_run
% fsp_validation_comment Disable firewall 
\begin{lstlisting}[language=bash,keywords={}]
[master]$ systemctl disable SuSEfirewall2
[master]$ systemctl stop SuSEfirewall2
\end{lstlisting}
% end_fsp_run

\subsection{Add resource management services on {\em master} node} \label{sec:add_rm}
\input{../../common/install_slurm}

\subsection{Add \InfiniBand{} support services on {\em master} node} \label{sec:add_ofed}

The following commands add OFED and PSM support using base distro-provided drivers to the
chosen {\em master} host. Note that the default permission for InfiniPath
devices under \baseOS{} are not directly accessible by users. Consequently, the commands
below include the addition of \texttt{udev} rules that are provided by \FSP{}
to enable user access.

% begin_fsp_run
% fsp_comment_header Add InfiniBand support services on master node \ref{sec:add_ofed}
\begin{lstlisting}[language=bash,keywords={}]
[master]$ (*\install*) libibverbs-runtime libmlx4-rdmav2 libipathverbs-rdmav2
[master]$ (*\install*) libibmad5 librdmacm1 rdma infinipath-psm dapl-devel dapl-utils 

# Provide udev rules to enable /dev/ipath access for InfiniPath devices
[master]$ cp /opt/fsp/pub/examples/udev/60-ipath.rules /etc/udev/rules.d/

# Load IB drivers
[master]$ service rdma start
\end{lstlisting}
% end_fsp_run

With the \InfiniBand{} drivers included, you can also enable IPoIB functionality
which provides a mechanism to send IP packets over the IB network. If you plan
to mount a \Lustre{} file system over \InfiniBand{} (see \S\ref{sec:lustre_client}
for additional details), then having IPoIB enabled is a requirement for the
\Lustre{} client. \FSP{} provides a template configuration file to aid in setting up
an {\em ib0} interface on the {\em master} host. To use, copy the template
provided and update the \texttt{\$\{master\_ipoib\}} and
\texttt{\$\{ipoib\_netmask\}} entries to match local desired settings (alter ib0
naming as appropriate if system contains dual-ported or multiple HCAs). 

% begin_fsp_run
% fsp_validation_newline
% fsp_command if [ ${enable_ipoib} -eq 1 ];then
% fsp_indent 5
% fsp_validation_comment Enable ib0
\begin{lstlisting}[language=bash,literate={-}{-}1,keywords={},upquote=true]
[master]$ cp /opt/fsp/pub/examples/network/sles/ifcfg-ib0 /etc/sysconfig/network

# Define local IPoIB address and netmask
[master]$ perl -pi -e "s/master_ipoib/${sms_ipoib}/" /etc/sysconfig/network/ifcfg-ib0
[master]$ perl -pi -e "s/ipoib_netmask/${ipoib_netmask}/" /etc/sysconfig/network/ifcfg-ib0

# Initiate ib0
[master]$ ifup ib0
\end{lstlisting}
% fsp_indent 0
% fsp_command fi
% end_fsp_run

\subsection{Add Cluster Checker} \label{sec:add_clck}
\input{../../common/cluster_checker}

\subsection{Complete basic Warewulf setup for {\em master} node} \label{sec:setup_ww}
\input{../../common/warewulf_setup}

% begin_fsp_run
% fsp_comment_header Complete basic Warewulf setup for master node \ref{sec:setup_ww}
%\begin{verbatim}

\begin{lstlisting}[language=bash,literate={-}{-}1,keywords={},upquote=true,keepspaces]
# Configure Warewulf provisioning to use desired internal interface
[master]$ perl -pi -e "s/device = eth1/device = ${sms_eth_internal}/" /etc/warewulf/provision.conf

# Configure DHCP server to use desired internal interface
[master]$ perl -pi -e "s/^DHCPD_INTERFACE=\S+/DHCPD_INTERFACE=${sms_eth_internal}/" /etc/sysconfig/dhcpd

# Enable tftp service for compute node image distribution
[master]$ perl -pi -e "s/^\s+disable\s+= yes/ disable = no/" /etc/xinetd.d/tftp

# Configure Warewulf to use the default SLES tftp location
[master]$ perl -pi -e "s#\#tftpdir = /var/lib/#tftpdir = /srv/#" /etc/warewulf/provision.conf

# Update Warewulf http configuration to use the SLES version of mod_perl
[master]$ export MODFILE=/etc/apache2/conf.d/warewulf-httpd.conf
[master]$ perl -pi -e "s#modules/mod_perl.so\$#/usr/lib64/apache2/mod_perl.so#" $MODFILE

# Enable http access for Warewulf cgi-bin directory to support newer apache syntax
[master]$ perl -pi -e "s/cgi-bin>\$/cgi-bin>\n Require all granted/" $MODFILE
[master]$ perl -pi -e "s/Allow from all/Require all granted/" $MODFILE
[master]$ perl -ni -e "print unless /^\s+Order allow,deny/" $MODFILE

# Enable internal interface for provisioning
[master]$ ifconfig ${sms_eth_internal} ${master_ip} netmask ${internal_netmask} up

# Restart/enable relevant services to support provisioning
[master]$ service xinetd restart
[master]$ systemctl enable mysql.service
[master]$ service mysql restart                          
[master]$ systemctl enable apache2.service
[master]$ service apache2 restart
\end{lstlisting}
%\end{verbatim}
% end_fsp_run

\subsection{Define {\em compute} image for provisioning}

With the provisioning services enabled, the next step is to define and
customize a system image that can subsequently be used to provision one or more
{\em compute} nodes. The following subsections highlight this process.

\subsubsection{Build initial BOS image} \label{sec:assemble_bos}

The \FSP{} build of \Warewulf{} includes specific enhancements enabling support for
\baseOS{}. The following steps illustrate the process to build a minimal, default
image for use with \Warewulf{}.  We begin by creating a directory structure on the 
{\em master} that will represent the root filesystem of the compute node. The 
default location for this example is in \path{/opt/fsp/admin/images/sles12}.

% begin_fsp_run
% fsp_comment_header Create compute image for Warewulf \ref{sec:assemble_bos}
\begin{lstlisting}[language=bash,literate={-}{-}1,keywords={},upquote=true,keepspaces]
# Define chroot location for SLES 
[master]$ export CHROOT=/opt/fsp/admin/images/sles12

# Build initial chroot image
[master]$ mkdir -p -m 755 $CHROOT                        # create chroot housing dir
[master]$ mkdir -m 755 $CHROOT/dev                       # create chroot /dev dir
[master]$ mknod -m 666 $CHROOT/dev/zero c 1 5            # create /dev/zero device
[master]$ wwmkchroot sles-12 $CHROOT                     # create base image
\end{lstlisting}
% end_fsp_run

\subsubsection{Add \FSP{} components} \label{sec:add_components}

The \texttt{wwmkchroot} process used in the previous step is designed to
provide a minimal \baseOS{} configuration. Next, we add additional components to
include resource management client services, \InfiniBand{} drivers, and other
additional packages to support the default \FSP{} environment.  This process uses
the \texttt{chroot} command to augment the base provisioning image and will
access the BOS and \FSP{} repositories to resolve package install requests. To
access the remote repositories by hostname (and not IP addresses), the chroot
environment needs to be updated to enable DNS resolution. Assuming that
the {\em master} host has a working DNS configuration in place, the chroot environment can
be updated with a copy of the configuration as follows:

% begin_fsp_run
% fsp_comment_header Add FSP components to compute image \ref{sec:add_components}
\begin{lstlisting}[language=bash,literate={-}{-}1,keywords={},upquote=true]
[master]$ cp -p /etc/resolv.conf $CHROOT/etc/resolv.conf
\end{lstlisting}
% end_fsp_run

%\newpage
% begin_fsp_run
\begin{lstlisting}[language=bash,literate={-}{-}1,keywords={},upquote=true]
# Import GPG keys for chroot repository usage
[master]$ zypper -n --root $CHROOT --no-gpg-checks --gpg-auto-import-keys refresh

# Add SLURM client support
[master]$ (*\groupchrootinstall*) fsp-slurm-client

# Add IB support
[master]$ (*\chrootinstall*) libibverbs-runtime libmlx4-rdmav2 libipathverbs-rdmav2
[master]$ (*\chrootinstall*) libibmad5 librdmacm1 rdma infinipath-psm dapl-devel dapl-utils

# Provide udev rules to enable /dev/ipath access for InfiniPath devices
[master]$ cp /opt/fsp/pub/examples/udev/60-ipath.rules $CHROOT/etc/udev/rules.d/

# Add Network Time Protocol (NTP) support
[master]$ (*\chrootinstall*) ntp

# Include FSP default modules user environment
[master]$ (*\chrootinstall*) lmod-defaults-intel-fsp

# Enable ssh access 
[master]$ chroot $CHROOT systemctl enable sshd.service

# Remove default hostname to allow WW to provision network names
[master]$ mv $CHROOT/etc/hostname $CHROOT/etc/hostname.orig
\end{lstlisting}
% end_fsp_run

\subsubsection{Customize system configuration} \label{sec:master_customization}

Prior to assembling the image, it is advantageous to perform any additional
customizations within the chroot environment created for the desired {\em
  compute} instance. The following steps document the process to add a local
{\em ssh} key created by \Warewulf{} to support remote access, identify the
resource manager server, configure NTP for compute resources, and enable \NFS{}
mounting of a \$HOME file system and the public \FSP{} install path
(\texttt{/opt/fsp/pub}) that will be hosted by the {\em master} host in this
example configuration.  
%The \NFS{} exporting options use an address/netmask
%combination to limit the export scope to the defined compute nodes.

% begin_fsp_run
% fsp_comment_header Customize system configuration \ref{sec:master_customization}
\begin{lstlisting}[language=bash,literate={-}{-}1,keywords={},upquote=true]
# add new cluster key to base image
[master]$ cat ~/.ssh/cluster.pub >> $CHROOT/root/.ssh/authorized_keys

# add NFS client mounts of /home and /opt/fsp/pub to base image
[master]$ echo "${master_ip}:/home /home nfs nfsvers=3,rsize=1024,wsize=1024,cto 0 0" >> $CHROOT/etc/fstab
[master]$ echo "${master_ip}:/opt/fsp/pub /opt/fsp/pub nfs nfsvers=3 0 0" >> $CHROOT/etc/fstab

# Identify resource manager server IP in SLURM configuration file on computes
[master]$ perl -pi -e "s/ControlMachine=\S+/ControlMachine=${master_ip}/" $CHROOT/etc/slurm/slurm.conf

# Identify resource manager hostname on master host
[master]$ perl -pi -e "s/ControlMachine=\S+/ControlMachine=${master_name}/" /etc/slurm/slurm.conf

# Export /home and FSP public packages from master server
[master]$ echo "/home *(rw,no_subtree_check,fsid=10,no_root_squash)" >> /etc/exports
[master]$ echo "/opt/fsp/pub *(ro,no_subtree_check,fsid=11)" >> /etc/exports
[master]$ exportfs -a
[master]$ service nfsserver restart

# Enable NTP time service on computes and identify master host as local NTP server
[master]$ chroot $CHROOT systemctl enable ntpd
[master]$ echo "server ${master_ip}" >> $CHROOT/etc/ntp.conf
\end{lstlisting}
% end_fsp_run

% Additional commands when additional computes are requested

% begin_fsp_run
% fsp_validation_newline
% fsp_validation_comment Update basic slurm configuration if additional computes defined
% fsp_command if [ ${num_computes} -gt 4 ];then
% fsp_command    perl -pi -e "s/^NodeName=(\S+)/NodeName=c[1-${num_computes}]/" /etc/slurm/slurm.conf
% fsp_command    perl -pi -e "s/^PartitionName=normal Nodes=(\S+)/PartitionName=normal Nodes=c[1-${num_computes}]/" /etc/slurm/slurm.conf

% fsp_command    perl -pi -e "s/^NodeName=(\S+)/NodeName=c[1-${num_computes}]/" $CHROOT/etc/slurm/slurm.conf
% fsp_command    perl -pi -e "s/^PartitionName=normal Nodes=(\S+)/PartitionName=normal Nodes=c[1-${num_computes}]/" $CHROOT/etc/slurm/slurm.conf
% fsp_command fi
% end_fsp_run

\subsubsection{Additional Customizations ({\em optional})} \label{sec:addl_customizations}

This section highlights common additional customizations that
can {\em optionally} be applied to the
local cluster environment. These customizations include:

\begin{itemize*}
\item Increase memlimits to support MPI jobs over \InfiniBand{}
\item Restrict ssh access to compute resources
\item Add \Lustre{} client
\end{itemize*}

\noindent Details on the steps required for each of these customizations are
discussed further in the following sections.

\paragraph{Increase locked memory limits}

\input{../../common/memlimits} 

% begin_fsp_run
% fsp_comment_header Additional customizations \ref{sec:addl_customizations}
\begin{lstlisting}[language=bash,keywords={},upquote=true]
# Update memlock settings on master
[master]$ echo "* soft memlock unlimited" >> /etc/security/limits.conf
[master]$ echo "* hard memlock unlimited" >> /etc/security/limits.conf

# Update memlock settings within compute image
[master]$ echo "* soft memlock unlimited" >> $CHROOT/etc/security/limits.conf
[master]$ echo "* hard memlock unlimited" >> $CHROOT/etc/security/limits.conf
\end{lstlisting}
% end_fsp_run

%# Enable larger locked memory limits for SLURM daemon
%[master]$ echo "ulimit -l unlimited" >> $CHROOT/etc/sysconfig/slurm

\paragraph{Enable ssh control via resource manager} 

\input{../../common/slurm_pam}

% begin_fsp_run
% fsp_validation_comment Enable slurm pam module
\begin{lstlisting}[language=bash,keywords={},upquote=true]
[master]$ echo "account    required     pam_slurm.so" >> $CHROOT/etc/pam.d/sshd
\end{lstlisting}
% end_fsp_run

\paragraph{Add \Lustre{} client} \label{sec:lustre_client}

\input{../../common/lustre-client}
% SLES-specific addition
Additionally, note that a default \baseOS{} environment may not allow loading of
the necessary \Lustre{} kernel modules. Consequently, the example below includes
steps which update the {\texttt /etc/modprobe.d/10-unsupported-modules.conf}
file to allow loading of the necessary modules.

% begin_fsp_run
% fsp_validation_newline
% fsp_command if [ ${enable_lustre_client} -eq 1 ];then
% fsp_indent 5

% fsp_validation_comment Install Lustre client on master
\begin{lstlisting}[language=bash,keywords={},upquote=true]
# Add Lustre client software to master host
[master]$ (*\install*) lustre-client-fsp lustre-client-fsp-modules

# Update configuration to allow Lustre modules to be loaded on master host
[master]$ perl -pi -e "s/^allow_unsupported_modules 0/allow_unsupported_modules 1/" \
     /etc/modprobe.d/10-unsupported-modules.conf
\end{lstlisting}
% end_fsp_run

\clearpage
% begin_fsp_run
% fsp_validation_comment Enable lustre in WW compute image
\begin{lstlisting}[language=bash,keywords={},upquote=true]
# Include Lustre client software in compute image
[master]$ (*\chrootinstall*) lustre-client-fsp lustre-client-fsp-modules

# Update configuration to allow Lustre modules to be loaded on compute hosts
[master]$ perl -pi -e "s/^allow_unsupported_modules 0/allow_unsupported_modules 1/" \
     $CHROOT/etc/modprobe.d/10-unsupported-modules.conf

# Include mount point and file system mount in compute image
[master]$ mkdir $CHROOT/mnt/lustre
[master]$ echo "${mgs_fs_name} /mnt/lustre lustre defaults,_netdev,localflock 0 0" >> $CHROOT/etc/fstab
\end{lstlisting}
% end_fsp_run



The default underlying network type used by \Lustre{} is {\em tcp}. If your
external \Lustre{} file system is to be mounted using a network type other than
tcp, additional configuration files are necessary to identify the desired
network type. The example below illustrates creation of modprobe configuration files
instructing \Lustre{} to use an \InfiniBand{} network with the {\em o2ib} LNET driver
attached to \texttt{ib0}. Note that these modifications are made to both the
{\em master} host and {\em compute} image.

% begin_fsp_run
% fsp_validation_comment Enable o2ib for Lustre
\begin{lstlisting}[language=bash,keywords={},upquote=true]
[master]$ echo "options lnet networks=o2ib(ib0)" >> /etc/modprobe.d/lustre.conf
[master]$ echo "options lnet networks=o2ib(ib0)" >> $CHROOT/etc/modprobe.d/lustre.conf
\end{lstlisting}
% end_fsp_run

With the \Lustre{} configuration complete, the client can be mounted on the {\em master}
host as follows:
% begin_fsp_run
% fsp_validation_comment mount Lustre client on master
\begin{lstlisting}[language=bash,keywords={},upquote=true]
[master]$ mkdir /mnt/lustre
[master]$ mount -t lustre -o localflock ${mgs_fs_name} /mnt/lustre
\end{lstlisting}
% fsp_indent 0
% fsp_command fi
% end_fsp_run

%% [FSP-469]: disabling ORCM

%% [FSP-469] \paragraph{Enable ORCM RAS subsystem} 
%% [FSP-469] \input{../../common/orcm}
%% [FSP-469] 
%% [FSP-469] % begin_fsp_run
%% [FSP-469] % fsp_validation_comment Configure ORCM aggregator and DB
%% [FSP-469] \begin{lstlisting}[language=bash,keywords={},upquote=true,keepspaces]
%% [FSP-469] # Set master host as data aggregator
%% [FSP-469] [master]$ perl -pi -e "s/aggregator_hostname/localhost/" /opt/open-rcm/etc/orcm-site.xml
%% [FSP-469] [master]$ perl -pi -e "s/Servername = (<.+>)/Servername = ${master_ip}/" /etc/orcmdb_psql.ini
%% [FSP-469] 
%% [FSP-469] # Update BMC credentials to match local values
%% [FSP-469] 
%% [FSP-469] [master]$ perl -pi -e "s/your-bmc-username/<bmc_username>/" /opt/open-rcm/etc/openmpi-mca-params.conf
%% [FSP-469] [master]$ perl -pi -e "s/your-bmc-password/<bmc_password>/" /opt/open-rcm/etc/openmpi-mca-params.conf
%% [FSP-469] 
%% [FSP-469] # Initialize postgres DB driver
%% [FSP-469] [master]$ odbcinst -i -d -f /etc/psql_odbc_driver.ini
%% [FSP-469] [master]$ odbcinst -i -s -h -f /etc/orcmdb_psql.ini 
%% [FSP-469] 
%% [FSP-469] # Initialize postgres
%% [FSP-469] [master]$ service postgresql start   
%% [FSP-469] 
%% [FSP-469] # Update postgres configuration files using ORCM templates
%% [FSP-469] [master]$ cp /etc/postgresql.orcm.conf /var/lib/pgsql/data/postgresql.conf
%% [FSP-469] [master]$ cp /etc/pg_hba.orcm.conf /var/lib/pgsql/data/pg_hba.conf
%% [FSP-469] 
%% [FSP-469] # Restart postgres
%% [FSP-469] [master]$ service postgresql restart 
%% [FSP-469] 
%% [FSP-469] # Create ORCM DB
%% [FSP-469] [master]$ sudo -u postgres psql --command "CREATE USER orcmuser WITH SUPERUSER PASSWORD 'orcmpassword';"
%% [FSP-469] [master]$ sudo -u postgres createdb -O orcmuser orcmdb
%% [FSP-469] [master]$ sudo -u postgres psql --username=orcmuser --dbname=orcmdb -f /etc/orcmdb_psql.sql
%% [FSP-469] 
%% [FSP-469] # Start ORCM on master host (requires both orcmsched and orcmd)
%% [FSP-469] [master]$ systemctl start orcmsched
%% [FSP-469] [master]$ systemctl start orcmd
%% [FSP-469] \end{lstlisting}
%% [FSP-469] % end_fsp_run
%% [FSP-469] 
%% [FSP-469] \input{../../common/orcm_client}

\subsubsection{Import files} \label{sec:file_import}

The \Warewulf{} system includes functionality to import arbitrary files from the
provisioning server for distribution to managed hosts. This is one way
to distribute user credentials to {\em compute} hosts. To
import local file-based credentials, issue the following:

% begin_fsp_run
% fsp_comment_header Import files \ref{sec:file_import}
\begin{lstlisting}[language=bash,literate={-}{-}1,keywords={},upquote=true]
[master]$ wwsh file import /etc/passwd                                                          
[master]$ wwsh file import /etc/group
[master]$ wwsh file import /etc/shadow 
\end{lstlisting}
% \end_fsp_run

Similarly, to import the cryptographic key that is required by the {\em munge}
authentication library to be available on every host in the resource management
pool, issue the following:

% begin_fsp_run
\begin{lstlisting}[language=bash,literate={-}{-}1,keywords={},upquote=true]
[master]$ wwsh file import /etc/munge/munge.key
\end{lstlisting}
% \end_fsp_run

Finally, to add support for controlling IPoIB interfaces, \FSP{} includes a
template file for \Warewulf{} that can optionally be imported and used later to provision
\texttt{ib0} network settings.

% begin_fsp_run
% fsp_validation_newline
% fsp_command if [ ${enable_ipoib} -eq 1 ];then
% fsp_indent 5
\begin{lstlisting}[language=bash,literate={-}{-}1,keywords={},upquote=true]
[master]$ wwsh file import /opt/fsp/pub/examples/network/sles/ifcfg-ib0.ww
[master]$ wwsh -y file set ifcfg-ib0.ww --path=/etc/sysconfig/network/ifcfg-ib0
\end{lstlisting}
% fsp_indent 0
% fsp_command fi
% \end_fsp_run

\input{../../common/finalize_provisioning}

% begin_fsp_run
% fsp_validation_comment Add hosts to cluster

\begin{lstlisting}[language=bash,keywords={},upquote=true,basicstyle=\footnotesize\ttfamily]
# Define four compute nodes and network settings 
[master]$ wwsh -y node new ${c_name[0]} --ipaddr=${c_ip[0]} --hwaddr=${c_mac[0]} -D ${eth_provision}
[master]$ wwsh -y node new ${c_name[1]} --ipaddr=${c_ip[1]} --hwaddr=${c_mac[1]} -D ${eth_provision}
[master]$ wwsh -y node new ${c_name[2]} --ipaddr=${c_ip[2]} --hwaddr=${c_mac[2]} -D ${eth_provision}
[master]$ wwsh -y node new ${c_name[3]} --ipaddr=${c_ip[3]} --hwaddr=${c_mac[3]} -D ${eth_provision}

# Optionally register additional compute nodes
[master]$ for ((i=4; i<$num_computes; i++)) ; do
              wwsh -y node new ${c_name[$i]} --ipaddr=${c_ip[$i]} --hwaddr=${c_mac[$i]} -D ${eth_provision}
           done

# Define provisioning image for hosts
[master]$ wwsh -y provision set "${compute_regex}" --vnfs=sles12 --bootstrap=`uname -r` \
     --files=dynamic_hosts,passwd,group,shadow,munge.key 
\end{lstlisting}

% fsp_validation_newline
% fsp_validation_comment Optionally, define IPoIB network settings (required if planning to mount Lustre over IB)
% fsp_command if [ ${enable_ipoib} -eq 1 ];then
% fsp_indent 5
\begin{lstlisting}[language=bash,keywords={},upquote=true,basicstyle=\footnotesize\ttfamily]
# Optionally define IPoIB network settings (required if planning to mount Lustre over IB)
[master]$ for ((i=0; i<$num_computes; i++)) ; do
              wwsh -y node set ${c_name[$i]} -D ib0 --ipaddr=${c_ipoib[$i]} --netmask=${ipoib_netmask}
           done
[master]$ wwsh -y provision set "${compute_regex}" --fileadd=ifcfg-ib0.ww
\end{lstlisting}
% fsp_indent 0
% fsp_command fi
% fsp_validation_newline

\clearpage
\begin{lstlisting}[language=bash,keywords={},upquote=true,basicstyle=\footnotesize\ttfamily]
# Restart dhcp / update PXE
[master]$ service dhcpd restart
[master]$ wwsh pxe update
\end{lstlisting}
% end_fsp_run

% begin_fsp_run

% fsp_validation_comment Optionally, add arguments to bootstrap kernel
% fsp_command if [ ${enable_kargs} ]; then
% fsp_command    wwsh provision set "${compute_regex}" --kargs=${kargs}
% fsp_command fi

% end_fsp_run

\subsection{Boot compute nodes} \label{sec:boot_computes}

\input{../../common/reset_computes} 
The following commands use the \texttt{ipmitool} utility to initiate power
resets on each of the four compute hosts. Note that the utility requires that
the \texttt{IPMI\_PASSWORD} environment variable be set with the local BMC password in
order to work interactively.

% begin_fsp_run
% fsp_comment_header Boot compute nodes \ref{sec:boot_computes}

\begin{lstlisting}[language=bash,keywords={},upquote=true]
[master]$ ipmitool -E -I lanplus -H ${c_bmc[0]} -U root chassis power reset   # power cycle c1
[master]$ ipmitool -E -I lanplus -H ${c_bmc[1]} -U root chassis power reset   # power cycle c2
[master]$ ipmitool -E -I lanplus -H ${c_bmc[2]} -U root chassis power reset   # power cycle c3
[master]$ ipmitool -E -I lanplus -H ${c_bmc[3]} -U root chassis power reset   # power cycle c4
\end{lstlisting} 

Once kicked off, the boot process should take less than 5 minutes (depending on
BIOS post times) and you can verify that the compute hosts are available via
ssh, or via parallel ssh tools to multiple hosts. For example, to run a command
on the newly imaged compute hosts using \texttt{pdsh}, execute the following:

\begin{lstlisting}[language=bash]
[master]$ pdsh -w c[1-4] uptime
c1  05:03am  up   0:02,  0 users,  load average: 0.20, 0.13, 0.05
c2  05:03am  up   0:02,  0 users,  load average: 0.20, 0.14, 0.06
c3  05:03am  up   0:02,  0 users,  load average: 0.19, 0.15, 0.06
c4  05:03am  up   0:02,  0 users,  load average: 0.15, 0.12, 0.05
\end{lstlisting}



\section{Install \FSP{} Development Components}

The install procedure outlined in \S\ref{sec:basic_install}
highlighted the steps necessary to install a {\em master} host,
assemble and customize a {\em compute} image, and provision several
compute hosts from bare-metal.  With these steps completed, 
%a {\em compute} Once the completion of the
%basic install procedure outlined in Section~\ref{sec:basic_install} is
%complete, 
additional \FSP{}-provided packages can now be added to support a flexible HPC
development environment including development tools, C/C++/Fortran compilers,
MPI stacks, and a variety of 3rd party libraries. The following subsections
highlight the additional software installation procedures, including the
addition of Intel licensed software (e.g. Composer compiler suite, \Intel{}
MPI). It is assumed that the end-site administrator will procure and install
the necessary licenses in order to use the Intel proprietary software.

\subsection{Development Tools} \label{sec:install_dev_tools}
\input{../../common/dev_tools}

\subsection{Compilers} \label{sec:install_compilers}
\input{../../common/compilers}

\subsection{Performance Tools} \label{sec:install_perf_tools}
\input{../../common/perf_tools}

\subsection{MPI Stacks} \label{sec:mpi}
\input{../../common/mpi}

\subsection{3rd Party Libraries and Tools} \label{sec:3rdparty}

\FSP{} provides pre-packaged builds for a number of popular open-source
tools and libraries used by HPC applications and developers. For
example, \FSP{} provides builds for \FFTW{} and \hdffive{} (including serial and parallel
I/O support), and the \GNU{} Scientific Library (GSL). Again, multiple builds of
each package are available in the \FSP{} repository to support multiple compiler
and MPI family combinations where appropriate. The general naming convention
for builds provided by \FSP{} is to append the compiler and MPI family name that
the library was built against directly into the package name. For example,
libraries that do not require MPI as part of the build process adopt the
following RPM name: \\

\noindent
\texttt{package-<compiler\_family>-fsp-<package\_version>-<release>.rpm} \\

\noindent Packages that require MPI as part of the build expand upon this convention to
additionally include the MPI family name as follows: \\

\noindent
\texttt{package-<compiler\_family>-<mpi\_family>-fsp-<package\_version>-<release>.rpm} \\

To illustrate this further, the command below queries the locally configured
repositories to identify all of the available \FFTW{} packages that were built
with the \GNU{} toolchain. The resulting output that is included shows that
pre-built versions are available for each of the supported MPI families
presented in \S\ref{sec:mpi}.

\begin{lstlisting}[language=bash]
[master]$ zypper search -t package fftw-gnu-*-fsp
Loading repository data...
Reading installed packages...

S | Name                  | Summary                          | Type   
--+-----------------------+----------------------------------+--------
  | fftw-gnu-impi-fsp     | A Fast Fourier Transform library | package
  | fftw-gnu-mvapich2-fsp | A Fast Fourier Transform library | package
  | fftw-gnu-openmpi-fsp  | A Fast Fourier Transform library | package
\end{lstlisting}

Note that \FSP{}-provided 3rd party builds are configured to be installed
into a common top-level repository so that they can be easily exported to
desired hosts within the cluster. This common top-level path
(\texttt{/opt/fsp/pub}) was previously configured to be mounted on {\em
  compute} nodes in \S\ref{sec:master_customization}, so the packages will be
immediately available for use on the cluster after installation on the {\em
  master} host.  For convenience, \FSP{} provides package aliases for these 3rd
party libraries and utilities that can be used to install all of the available
compiler/MPI family permutations. To install all of the available package
offerings within \FSP{}, issue the following:

\input{../../common/third_party_libs}

\section{Resource Manager Startup} \label{sec:rms_startup}
\input{../../common/slurm_startup}

\section{Run a Test Job} \label{sec:test_job}
\input{../../common/slurm_test_job}

\clearpage
\appendix
\section*{Appendices}
\addcontentsline{toc}{section}{Appendices}
\renewcommand{\thesubsection}{\Alph{subsection}}

\input{../../common/automation_appendix}
\input{manifest}
\input{../../common/signature}

\end{document}

